The present invention relates to an image processing apparatus configured to generate a high resolution image having higher resolution than low resolution images from the low resolution images, a method of controlling the image processing apparatus, and a non-transitory computer-readable recording medium.
Conventionally, super resolution techniques are known as image processing techniques for causing an image having low resolution to have high resolution. A multiframe super resolution technique is known as one of the super resolution techniques. In the multiframe super resolution technique, a plurality of low resolution images related to the same target is used to generate a single high resolution image.
More specifically, resolution increase processing is carried out by using low resolution images acquired by shifting a position of a subject or the like little by little to perform imaging so that a single high resolution image is generated.
The increase in the resolution is carried out on the basis of a reference image selected from the low resolution images. Specifically, an amount of a positional shift between the low resolution images is first calculated based on the reference image and the resolution increase processing is performed on the basis of the positional shift amount thus calculated and image data on respective low resolution images.
In many cases, a first one of the low resolution images read in order is selected as the reference image. However, a content of the high resolution image obtained by the increase in the resolution (which includes image quality and a composition) greatly depends on the reference image. Accordingly, a result of the high resolution image which is obtained differs greatly between the case in which a frame of the low resolution images which has the highest image quality is selected as the reference image and the case in which a frame of the low resolution images which has the lowest image quality is selected as the reference image.
Specific description will be given with reference to FIGS. 21A and 21B as follows. FIGS. 21A and 21B illustrate the case in which a single high resolution image is generated by using three low resolution images. FIG. 21A shows the case in which a low resolution image LP101 having the highest image quality in three low resolution images LP101, LP102 and LP103 is selected as the reference image. In the case shown in FIG. 21A, an excellent high resolution image HP101 is obtained.
On the other hand, FIG. 21B shows the case in which the low resolution image LP103 having the lowest image quality in the three low resolution images LP101, LP102 and LP103 is selected as the reference image. In the case shown in FIG. 21B, a high resolution image HP102 thus obtained is unclear.
Therefore, there is conventionally made a proposal such as selection of a reference image based on a small blur amount of a whole screen (Japanese Unexamined Patent Publication No. 2009-194896 (published on Aug. 27, 2009)).
In the field of the super resolution technique, there is further made a proposal for calculating a blur amount of each area on an image and carrying out super resolution processing in a parameter corresponding to the blur amount of every area in order to enhance image quality (Japanese Unexamined Patent Publication No. 2010-73074 (published on Apr. 2, 2010)).
Even if the reference image is selected based on the value of the blur amount of the whole image such as a small blur amount as proposed in the related art, however, an intended high resolution image may not be obtained in some cases. Specific description will be given with reference to FIGS. 22A and 22B as follows.
FIGS. 22A and 22B show two images in almost equal blur amounts of the whole image. FIG. 22A shows an image LP111 in which a person M100 is clear and a background B100 is blurred. Moreover, FIG. 22B shows an image LP112 in which the person M100 is blurred and the background B100 is clear.
Although the images LP111 and LP112 are images having almost equal blur amounts as the whole image, a natural high resolution image can be obtained from the image LP111 after the super resolution processing when they are set to be the reference images. In other words, it is possible to obtain a high resolution image having the clearer person M100 in the case in which the image LP111 having the clear person M100 is set to be the reference image than the case in which the image LP112 is set to be the reference image.
In the related art, accordingly, there is a problem in that an intended high resolution image may not be always obtained through the resolution processing by using a reference image corresponding to a target area.